Evaluation of the Efficiency of Instructional Design in Online Education: Data Mining of Course Lecture Video Usages by Students

Judy Luo, Ph.D.
Ying-Ying Kuo, Ed.D. 
Division of Instructional Technology
George Mason University
Oct. 22, 2014

Background

Online education has become popular since 2000.

Lecture videos have become indispensable components in online learning

Educators are interested in:

What are students’ study strategies when they study in a video-oriented course?

How do videos promote students’ learning?

Are students’ learning outcomes associated with their use of videos?

About the Courses

•  100% fully online course deployed in Blackboard Learn 9.1 (Learning Managment System)

•  An environmental science and policy course

•  A 3 credit, general required non-lab science course for non-science majors 

•  The course objective is to study the influence of human activities on the environment and the methodologies used to ameliorate those impacts.

Course Design

•  Five units including 16 lessons and 56 lecture videos

•  Students have access to all lectures at the beginning of the semester

•  5 graded unit quizzes, which can be taken as many times as needed

•  3 in-class exams

•  20 lesson activities and assignments

•  One research project

Research Questions

What were the students’ study strategies?

How was the students’ learning behavior associated with course design?

Did lecture videos support students’ learning and affect students’ learning outcomes?

Data Collection

All student activities during the time frame (Spring semester, 2014 at George Mason University, Fairfax, Virginia)

Number of Student Participants:  11

A course evaluation survey at the end of the semester to collect:

Learning motivation

Study strategies

Learning satisfaction

Learning experience

Data Collection & Analysis

Real video usage on Blackboard Learn LMS
–     Total entry (10290 rows of activities for students)   
–     Total count for video usage: (628)
–     Valid count for video usage estimation: (539)
–     Total video watching time estimation:  (351015 seconds)

Survey
–     Survey deliver time: May 2, 2014 (administrated in class)
–     Respond rate: 68.75%
–     Demographics	
             - 3 male students, 8 female students
             - 4 full time, 7 part time
             - 4 didn’t take any online course before

Advanced Reporting Database Schema Description

Advanced Reporting Database Schema Elements

SQL query to retrieve student activities in the course

Data Processing in Excel by Visual Basic Macro

Demo of Excel (Visual Basic)

Interested Variables Created in Excel

DURATION_SESSION

STUDENT_USE_CONTENT

DURATION

Week

Questions: Study Strategies

•	Did some students study lectures in advance of other students?

•	Did students follow the schedule to study?

•	Did students watch all the lecture videos?

•	How often did students use lecture videos?

•	Did watching videos affect students’ learning outcomes (satisfaction, grades, motivation)?

Data said: video use was associated with course schedule design

Course Schedule & Grade Component

Unit quizzes: (10%):  Unit 1 (Feb. 16); Unit 2 (March 9), Unit 3 (Apr. 06), Unit 4 (Apr. 27), Unit 5 (May 4)

Exams (40%):  Exam 1 (Mar 21), Exam 2 (Apr. 11), Exam 3 (May 9)

Research Paper (30%)	
•  Presentation (May 2)	
•  Paper project (Apr. 27)

Students’ learning behaviors

•  Majority of the students studied and watched video by following the schedule. 

•  Students’ intensive use of videos were associated with unit quizzes (1 and 3) and 3 exams.

•  Students’ did not watch videos during the paper presentation date and submission due date.

•  Some students skipped watching videos.  

•  Most of them took one or few attempts on unit quizzes.

Correspondence Analysis

Findings

Course design affected students’ use of videos

Students' grades was not associated with their use of videos

Students' motivation to learn was significantly associated with satisfaction and competency.

Students' study strategies on survey was not fully reflect their real use of videos on Bb

Students' satisfaction was significantly related to their final grades.

Conclusion

The video usage data of students are REAL data collected from Blackboard Learn backend Database.

The difficulties of data collection (customized database query, further data processing using programming logic, and missing data handling (can not be reasonably estimated))

Survey result actually can not reflect the real video usage

Students filled out the form based on memory subjectively.

Some random factors involved.

Future work:  data collect in future courses with the same course design but large sample size